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Why GAO Did This Study 

The Department of Homeland 
Security's (DHS) Bureau of 
Customs and Border Protection 
(CBP) is conducting a multiyear, 
multibillion-dollar acquisition of a 
new trade processing system 
planned to support the movement 
of legitimate imports and exports 
and strengthen border security. By 
congressional mandate, 
expenditure plans for this system, 
called the Automated Commercial 
Environment (ACE), must meet 
certain conditions, including GAO 
review. This study addresses the 
extent to which the latest plan, for 
fiscal year 2004, satisfies these 
conditions, provides information 
about DHS's efforts to implement 
GAO's recommendations for 
improving ACE management, and 
makes observations about ACE. 



What GAO Recommends 



To assist DHS in managing ACE 
and increasing the chances that 
future releases will deliver 
promised capabilities on time and 
within budget, GAO is making 
recommendations to the Secretary 
of Homeland Security aimed at 
addressing recurring cost and 
schedule problems. DHS concurred 
with GAO's recommendations and 
described specific actions that it is 
taking to respond to each. 



www.gao.gov/cgi-bin/getrpt7GAO-04-719 

To view the full product, including the scope 
and methodology, click on the link above. 
For more information, contact Randolph C. 
Hite at (202) 512-3439 or hiter@gao.gov. 



What GAO Found 

The DHS fiscal year 2004 ACE expenditure plan provides for certain 
activities, including system implementation infrastructure and support, 
operations and maintenance, and the definition and design of two future 
releases. This plan, including related program documentation and program 
officials' statements, largely satisfies the legislative conditions imposed by 
the Congress. Some of the recommendations that GAO has previously made 
to strengthen ACE management have been addressed, and DHS has 
committed to addressing those that remain. However, much remains to be 
done before these recommendations are fully implemented. Specifically, 
progress on overcoming human capital challenges has been slow. 

GAO made several observations about ACE. Specifically, the first two ACE 
releases, which have been deployed, are operating largely as intended. 
However, achieving this initial operating capability has been difficult. DHS 
has established a pattern of borrowing resources from future releases to 
address problems of near-term releases, which has adversely affected the 
cost, schedule, and capability commitments of ongoing releases. In 
particular, the delay in completing the second ACE release has introduced a 
pattern of increased reliance on concurrent activities to meet the dictated 
schedule, continued release schedule delays, and cost overruns for the 
ongoing releases, as the figure below indicates. This domino effect is 
continuing into Release 3 and beyond. This pattern is not likely to change 
unless the degree of concurrence among activities within and between 
releases is better controlled, which will require that the reasons for release 
quality problems that led to the concurrent activity be addressed. Until the 
reasons for this pattern are pinpointed and corrected, in part through 
implementation of GAO's unaddressed recommendations, DHS will not be 
positioned to deliver promised capabilities on time and within budget, or 
produce mission value commensurate with investment cost. 
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United States General Accounting Office 
Washington, D.C. 20548 



May 14, 2004 

The Honorable Thad Cochran 
Chairman 

The Honorable Robert C. Byrd 
Ranking Minority Member 
Subcommittee on Homeland Security 
Committee on Appropriations 
United States Senate 

The Honorable Harold Rogers 
Chairman 

The Honorable Martin Olav Sabo 
Ranking Minority Member 
Subcommittee on Homeland Security 
Committee on Appropriations 
House of Representatives 

In January 2004, U.S. Customs and Border Protection (CBP) within the 
Department of Homeland Security (DHS) submitted to the Congress its 
fiscal year 2004 expenditure plan, seeking release of $318.7 million for its 
Automated Commercial Environment (ACE) program. ACE is to be CBP's 
new trade system. The program's goals include facilitating the movement of 
legitimate trade through more effective trade account management and 
strengthening border security by identifying import and export 
transactions that have an elevated risk of posing a threat to the United 
States. As required by CBP's fiscal year 2004 appropriations, 1 we reviewed 
the expenditure plan. Our objectives were to (1) determine whether the 
ACE fiscal year 2004 expenditure plan satisfies certain legislative 
conditions, (2) determine the status of our open ACE recommendations, 
and (3) provide any other observations about the expenditure plan and 
DHS's management of the ACE program. 

On March 8, 2004, we briefed your offices on the results of this review. This 
report transmits the results of our work. The full briefing, including our 
scope and methodology, is reprinted as appendix I. 



^.L. 108-90 (October 1, 2003). 
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Compliance with 
Legislative Conditions 



The expenditure plan satisfied or partially satisfied the conditions specified 
in DHS's appropriations act. Specifically, the plan, including related 
program documentation and program officials' statements, satisfied or 
provided for satisfying all key aspects of (1) meeting the capital planning 
and investment control review requirements of the Office of Management 
and Budget (OMB), (2) complying with the DHS enterprise architecture, 2 
and (3) review and approval by DHS and OMB. The plan partially satisfied 
the condition that specifies compliance with the acquisition rules, 
requirements, guidelines, and systems acquisition management practices of 
the federal government. 3 



Status of Open 
Recommendations 



CBP is making progress in addressing our open recommendations. Each 
recommendation, along with the status of CBP's actions to address it, is 
summarized below. 



Before building each ACE release (i.e., beginning detailed design and 
development), certify to Customs' House and Senate appropriations 
subcommittees that the enterprise architecture has been sufficiently 
extended to provide the requisite enterprise design content, and has 
been updated to ensure consistency and integration across business 
areas. 



With respect to the fiscal year 2004 expenditure plan, CBP has completed 
action to implement this recommendation by using a process developed by 
the former Customs Service to certify ACE releases against the former 
Customs enterprise architecture. Further, DHS is developing a process to 
ensure alignment of ACE with the departmentwide enterprise architecture 
it is developing to replace DHS component agency and bureau 
architectures. 



2 An enterprise architecture is an institutional blueprint for guiding and constraining 
investments in business process change and systems. 

3 In June 2003, the Department of the Treasury's Office of Inspector General (OIG) issued a 
report on the ACE program's contract, concluding that the former Customs Service, now 
CBP, did not fully comply with Federal Acquisition Regulation requirements in the 
solicitation and award of the contract. CBP disagrees with the Treasury OIG conclusion. To 
resolve the disagreement, DHS asked us to render a formal decision. We are currently 
reviewing the matter. 
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• Develop and implement a rigorous and analytically verifiable cost- 
estimating program that embodied the tenets of effective estimating as 
defined in the Software Engineering Institute's (SEI) institutional 
and project-specific estimating guidance} 

CBP is in the process of implementing this recommendation. It has 
developed and begun to implement a cost-estimating program that 
embodies SEFs models. For example, CBP hired a contractor to define and 
implement a cost-estimating program and develop independent cost 
estimates. Additionally, it has defined and documented processes for 
estimating expenditure plan costs, and tasked contractors with ensuring 
that expenditure plan estimates are evaluated against SEI criteria for 
validating software cost and schedule estimates. 

• Immediately develop and implement a human capital management 
strategy that provides both near- and long-term solutions to program 
office human capital capacity limitations, and report quarterly to the 
appropriations committees on the progress of efforts to do so. These 
efforts should include defining the office's skill and capability needs 
in terms that will allow the program office to attract qualified 
individuals and that will provide sufficient rewards and training, 
linked to performance, to promote their retention. 

CBP is in the process of implementing this recommendation, and it has 
reported on these actions to the Congress. In particular, the program office 
developed and began implementing a human capital management plan that 
called for addressing several areas, including filling vacant positions. 
However, the program office has continued to experience difficulty in 
filling key positions and has begun implementing a new staffing plan 
intended to address DHS's concern that the program office has insufficient 
government program management staff. 



4 SEI's institutional and project-specific estimating guidelines are defined in Robert E. Park, 
Checklists and Criteria for Evaluating the Cost and Schedule Estimating Capabilities of 
Software Organizations, CMU/SEI-95-SR-005 (Tittsburgh, Pa.: Carnegie Mellon University 
Software Engineering Institute, 1995) and A Manager's Checklist for Validating Software 
Cost and Schedule Estimates, CMU/SEI-95-SR-004 (Tittsburg, Pa: 1995), respectively. 
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• Develop and implement process controls for SEFs Software 
Acquisition Capability Maturity Model (SA-CMM 5 ) level 2 key process 
areas and for the level 3 acquisition risk management key process 
area; develop and implement each of the missing SA-CMM key 
practices for these key process areas, and until this is accomplished, 
report to the appropriations committees quarterly on the progress of 
efforts to do so. 

CBP has implemented this recommendation. In November 2003, SEI 
assigned the program a level 2 rating, meaning that the program had 
established basic acquisition management processes. Further, CBP had 
addressed the two weaknesses SEI identified in the level 3 acquisition risk 
management area, and has reported to the Congress on the status of its 
acquisition management improvement activities. 

• Establish an independent verification and validation (IV&V) 
function to assist CBP in overseeing contractor efforts, such as 
testing? 

CBP is in the process of implementing this recommendation. The program 
office has established an IV&V function and has designated its IV&V 
contractor. However, program documentation describes roles for the IV&V 
contractor that are so integral to ACE development, testing, and 
deployment, as well as to ACE program management, that they raise 
questions about how the contractor could also perform independent ACE- 
related assessments. CBP officials stated that they plan to address this 
situation. 

• Have future ACE expenditure plans specifically address any proposals 
or plans, whether tentative or approved, for extending and using ACE 
infrastructure to support other homeland security applications, 
including any impact on ACE of such proposals and plans. 



Capability Maturity Model (CMM) is a service mark of Carnegie Mellon University and 
CMM is registered in the U.S. Patent and Trademark Office. The SA-CMM identifies key 
process areas that are necessary to effectively manage software-intensive system 
acquisitions. Level 2 is the second level of the SA-CMM's five-level scale; achieving this level 
means that an organization has the software acquisition rigor and discipline to repeat 
project successes. 

6 IV&V involves having an independent organization conduct unbiased reviews of 
management processes, products, and results with the goal of verifying and validating that 
these meet stated requirements and standards. 
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CBP plans to implement this recommendation. Program officials 
acknowledged the potential for ACE infrastructure to support other DHS 
system applications and the potential for integrating ACE data and 
applications with those systems. According to program officials, to begin 
preparing for this potential, they have focused initially on discussing 
collaboration opportunities with DHS's United States Visitor and Immigrant 
Status Indicator Technology (US-VISIT) program. 7 



Observations on 
Management of ACE 



We recognize accomplishments to date, including the deployment of the 
first 2 of 10 planned ACE releases, and address the need for rigorous and 
disciplined program management practices relating to managing ACE's 
costs, schedule, and requirements, as well as the degree of concurrent 
system development activities. An overview of the observations follows: 

• Release 1 and 2 testing revealed a sufficient volume and significance 
of system defects to affect schedule commitments. Release 1 testing 
revealed system defects, which were corrected about 3 months after the 
testing was concluded. Release 2 testing revealed more system defects 
than expected; addressing these defects required more time than 
originally planned for Release 2 testing. 

• Requirements planned for Releases 1 and 2 have been deferred to later 
releases. Releases 1 and 2 were originally planned as a single release 
that would satisfy 465 requirements. Subsequently, 103 of these 
requirements, which included capabilities related to document 
management, data recovery, and security, were deferred to Releases 3 
and 4, and 14 were deferred to later releases, for a total of 117 deferred 
requirements. 

• Release 2 delays have set in motion a pattern of increased reliance on 
concurrent activities, continued release delays, and cost overruns. The 
time required to resolve defects resulted in overlap of Release 2 test 
phases. Delays in completing Release 2 increased its overlap with 
Release 3 and have contributed to Release 3 delays. Likewise, Release 3 
delays will increase overlap with Release 4 and could cause Release 4 
delays. 



7 US-VISIT is a governmentwide program to collect, maintain, and share information on 
foreign nationals in order to enhance national security and facilitate legitimate trade and 
travel while adhering to U.S. privacy laws and policies. 
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Finally, the increased concurrency and schedule delays resulted in 
Releases 1 through 4 costing more than planned. 8 

• Releases 1 and 2 are largely operating according to adjusted 
requirements, defect density has stabilized, and the system is mostly 
up and running when needed. Defects prevalent in Releases 1 and 2 
have stabilized, but nontrivial defects remain. Releases 1 and 2, 
however, are mostly available when needed. 

• Following problems with Release 1, steps were taken to avoid future 
problems, but the success of these actions is unclear As a result of 
Release 1 cost overruns and schedule delays, CBP directed its 
contractor to develop and implement a plan to improve program 
performance. The objective of this plan was to demonstrate serious, 
tangible, and measurable actions to resolve program issues. However, 
program officials could not point to metrics that demonstrated 
improved performance, and the cost to develop and implement the 
corrective action plan is unknown. 

• Initial releases represent a small fraction of the total ACE software. 
Releases 1, 2, 3, and 4 account for only about 14 percent of ACE 
software and are to be designed, developed, and deployed within about 
3 years. In contrast, the remaining six ACE releases constitute 86 
percent of ACE software but are to be designed, developed, and 
deployed within about 3-1/2 years. CBP officials stated that they are 
currently studying the extent to which off-the-shelf software can be 
employed in the remaining six releases. 

• The reliability of key cost estimates in the expenditure plan is unclear 
The cost estimates in the expenditure plan varied from independent cost 
estimates, and the derivation of the expenditure plan estimates only 
partially satisfied SEI guidance for assessing the effectiveness of 
processes used to derive cost estimates. 



8 Our March 8, 2004, briefing to the staffs of the Subcommittees on Homeland Security, 
Senate and House Committees on Appropriations, reported a $23.3 million overrun for 
Releases 1 and 2 in addition to a $36.2 million overrun for Releases 3 and 4. Subsequently, 
CBP's ACE development contractor revised the estimated overrun for Releases 3 and 4 to 
$46 million. 
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Conclusions ^ e fi sca l y ear 2004 ACE expenditure plan, along with program 

documentation and officials' commitments, largely satisfies the legislative 
conditions imposed by the Congress. Further, many of the 
recommendations that we have made to strengthen ACE management have 
been addressed, and CBP leadership has committed to addressing those 
that remain. However, much must still be done before these 
recommendations are fully implemented. Particularly, progress has been 
slow on overcoming human capital challenges. 

To CBP's credit, the first two ACE releases are operating largely as 
intended. Achieving this initial operating capability, however, has not 
occurred without difficulty, such as borrowing resources from future 
releases to overcome problems on near-term releases; similar difficulties 
that could affect cost, schedule, and capability commitments are being 
experienced on ongoing releases. This pattern is not likely to change unless 
the degree of concurrency among activities within and between releases is 
better controlled, and the underlying reasons for introducing this 
concurrency are addressed. While time and resources were invested in 
management improvements over a year ago to address root causes, the 
absence of meaningful measures for determining whether these 
investments will be successful, coupled with limited progress on some of 
our previous recommendations, makes CBP's chances of delivering future 
ACE release capabilities on time and within budget uncertain at best. 



Recommendations for 
Executive Action 



To assist CBP in managing ACE and increasing the chances that future 
releases will deliver promised capabilities on time and within budget, we 
recommend that the Secretary of Homeland Security, through the Under 
Secretary for Border and Transportation Security and the CBP 
Commissioner, direct the CBP Chief Information Officer to take the 
following actions: 

• Ensure the independence of its IV&V agent. 

• Ensure that future expenditure plans are based on cost estimates that 
are reconciled with independent cost estimates. 

• Reconsider the ACE acquisition schedule and cost estimates in light of 
early release problems, including these early releases' cascading effects 
on future releases and their relatively small size compared with later 
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releases and in light of the need to avoid the past levels of concurrency 
among activities within and between releases. 

• Define measures, and collect and use associated metrics, for 
determining whether prior and future program management 
improvements are successful. 

• Report quarterly to the House and Senate Committees on 
Appropriations on efforts to address the above, as well as on our 
previous but unaddressed recommendations. 



Agency Comments and 
Our Evaluation 



In written comments on a draft of this report signed by the director, DHS 
Bankcard Programs and GAO/OIG Liaison, DHS concurred with our 
recommendations and stated actions that it is taking or plans to take to 
implement each. DHS also provided recommendations for disclosing 
certain sensitive information included in our draft report. We modified this 
report in accordance with DHS's recommendations. DHS's comments are 
reprinted in appendix II. 



We are sending copies of this report to the Chairmen and Ranking Minority 
Members of other Senate and House committees and subcommittees that 
have authorization and oversight responsibilities for homeland security. We 
are also sending copies to the Secretary of Homeland Security, the Under 
Secretary for Border and Transportation Security, the CBP Commissioner, 
and the Director of OMB. In addition, the report will be available at no 
charge on the GAO Web site at http://www.gao.gov. 
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Should you or your offices have any questions on matters discussed in this 
report, please contact me at (202) 512-3439 or by e-mail at hiter@gao.gov. 
Other contacts and key contributors to this report are listed in appendix III. 




Randolph C. Hite 

Director, Information Technology Architecture 
and Systems Issues 
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Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 
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Information Technology: Early Releases of Customs 
Trade System Operating, but Pattern of Cost and 
Schedule Problems Needs to Be Addressed 



Briefing to the Staffs of the 

Subcommittees on Homeland Security, 

Senate and House Committees on Appropriations 



March 8, 2004 



This briefing has been modified to exclude sensitive material that the Bureau of Customs and Border 
Protection deemed to be acquisition-sensitive. 
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Introduction 



The Department of Homeland Security (DHS) Bureau of Customs and Border 
Protection (CBP), formed from the former U.S. Customs Service and other entities 
with border protection responsibility, is acquiring a new trade processing system, 
known as the Automated Commercial Environment (ACE). The goals of ACE are to 

• facilitate the movement of legitimate trade through more effective trade 
account management; 

• strengthen border security by identifying import/export transactions that have 
an elevated risk of posing a threat to the United States or of violating a trade 
law or regulation; and 

• provide a single system interface between the trade community 1 and the 
federal government, 2 known as the International Trade Data System (ITDS), 
and thereby reduce the data reporting burden placed on the trade community 
while also providing federal agencies with the data and various capabilities to 
support their respective international trade and transportation missions. 



1 Members of the trade community include importers and exporters, brokers and trade advisors, and carriers. 

2 Federal agencies include CBP and other federal agencies responsible for managing international trade and transportation processes. 
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Introduction 



The Department of Homeland Security Appropriations Act, 2004, 1 limits obligating 
any funds for ACE until DHS submits for approval to the House and Senate 
Committees on Appropriations a plan for expenditure that 

1 . meets the capital planning and investment control review requirements 
established by the Office of Management and Budget (OMB), including 
Circular A-11, part 3; 2 

2. complies with CBP's enterprise architecture; 

3. complies with the acquisition rules, requirements, guidelines, and systems 
acquisition management practices of the federal government; 

4. is reviewed and approved by the CBP Investment Review Board, 3 DHS, and 
OMB; and 

5. is reviewed by GAO. 

1 Pub. L 108-90 (Oct. 1,2003). 

2 OMB Circular A-1 1 establishes policy for planning, budgeting, acquisition, and management of federal capital assets. 

3 The purpose of the IRB is to integrate capital planning and investment control, budgeting, acquisition, and management of 
investments. It is also to ensure that spending on investments directly supports and furthers the mission and that this spending 
provides optimal benefits and capabilities to stakeholders and customers. 
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Introduction 



In the Department of Homeland Security Appropriations Act, the Congress 
appropriated approximately $318.7 million for the ACE program. 1 

DHS submitted its fiscal year 2004 expenditure plan for $318.7 million on January 
21 , 2004, to its House and Senate Appropriations Subcommittees on Homeland 
Security. 



1 Pub. L. 108-90 (Oct. 1, 2003). 
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Objectives 



Objectives 

As agreed, our objectives were to 

• determine whether the ACE fiscal year 2004 expenditure plan satisfies the 
legislative conditions, 

• determine the status of our open recommendations made on ACE, and 

• provide any other observations about the expenditure plan and DHS's 
management of the ACE program. 

We conducted our work at CBP headquarters and contractor facilities in the 
Washington, D.C., metropolitan area from January through March 2004 in 
accordance with generally accepted government auditing standards. Details of our 
scope and methodology are given in attachment 1 . 
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Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 



Results in brief: Objective 1 
Legislative Conditions 



Fiscal year 2004 ACE expenditure plan satisfaction of legislative conditions 



Legislative conditions 


Status 


1 . Meets the capital planning and investment control review requirements 
established by OMB, including OMB Circular A-1 1 , part 7. a 


Satisfied 


2. Complies with DHS's enterprise architecture. 


Satisfied 


3. Complies with the acquisition rules, requirements, guidelines, and systems 
acquisition management practices of the federal government. 


Partially satisfied" 1 


4. Is reviewed and approved by the CBP Investment Review Board, DHS, and 
OMB. 


Satisfied 


5. Is reviewed by GAO. 


Satisfied 



Source: GAO. 

a Capital investment and control requirements are now found in OMB Circular A-1 1 , part 7, rather than part 3. 
b Satisfied or provides for satisfying every aspect of the condition that we reviewed. 

c As agreed with the staffs of the appropriations subcommittees, we assessed compliance with the DHS enterprise 
architecture in lieu of the CBP enterprise architecture. 

d Satisfied or provides for satisfying many, but not all, key aspects of the condition that we reviewed. 
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Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 



Results in brief: Objective 2 
Open Recommendations 



Status of actions to implement our open recommendations 



GAO recommendations 


Status 


Deiore DUiiumg eacn muc release, ceruiy to uor s appropriations 
subcommittees that the enterprise architecture has been sufficiently extended 
and updated. 


oompiete 


Develop and implement a rigorous and analytically verifiable cost estimating 


In progress 13 


program. 




Immediately develop and implement a human capital management strategy 


In progress 


that provides both near and long-term solutions; develop and implement 




missing human capital practices, and report quarterly to Congress on 




progress. 




Source: GAO. 




a Actions have been taken to fully implement the recommendation. 




b Actions are under way to implement the recommendation. 





8 




GAO 

Accountability * Integrity * Reliability 



Page 17 



GAO-04-719 Customs Modernization 



Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 



Results in brief: Objective 2 
Open Recommendations 



Status of actions to implement our open recommendations 



GAO recommendations Status 



Develop and implement process controls for Software Engineering Institute Complete 
Software Acquisition Capability Maturity Model 3 level 2 key process areas and 
for the level 3 acquisition risk management key process area; develop and 
implement missing practices, and report quarterly to Congress on progress. 

Establish an independent verification and validation function to assist CBP in In progress 
overseeing contractor efforts, such as testing. b 

Have future ACE expenditure plans specifically address any proposals or Planned 
plans for extending and using ACE infrastructure to support other homeland 
security applications. 

Source: GAO. 

a The Software Acquisition Capability Maturity Model® developed by Carnegie Mellon University's Software Engineering 
Institute (SEI) defines five levels of maturity that provide a roadmap for continuously improving an organization's acquisition 
management processes. 

independent Verification and Validation involves having an independent organization conduct unbiased reviews of test 
management processes, products, and results with the goal of verifying and validating that these meet stated requirements 
and standards. 

"Actions are planned to implement the recommendation. 
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Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 



Results in brief: Objective 3 
Observations 



Observations: 

• Release 1 and 2 testing revealed a sufficient volume and significance of 
system defects to impact schedule commitments. 

• Requirements planned for Releases 1 and 2 have been deferred. 

• Release 2 delays have set in motion a pattern of increased reliance on 
concurrent activities, continued release delays, and cost overruns. 

• Releases 1 and 2 are largely operating according to adjusted requirements; 
defect density has stabilized and system is mostly up and running when 
needed. 

• Following Release 1 problems, steps were taken to avoid future problems, but 
success of actions is unclear. 

• Initial releases represent a small fraction of the total ACE software. 

• Reliability of key cost estimates in expenditure plan is unclear. 
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Results in brief: Objective 3 
Observations 



To better ensure that future ACE releases deliver promised capability within budget 
and on time, we are making recommendations to the DHS Secretary to address a 
recurring pattern of cost and schedule problems. 

In their comments on a draft of this briefing, DHS, CBP, and ACE officials, including 
the DHS Chief Information Officer and the CBP Acting Chief Information Officer, 
generally agreed with the briefing and stated that it was fair and balanced. 
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ACE-Related Business Functions 
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ACE is to support CBP administration, enforcement, and trade compliance 
business operations. 

• Supported administration business operations include (1) processing and 
recording of revenue generated from trade compliance, and (2) maintaining 
the general ledger and accounting. 

• Supported enforcement business operations include identifying import and 
export transactions (i.e., cargo, conveyances, crew, and passengers) that 
have an elevated risk of threatening the United States or violating a trade law 
or regulation. 

• Supported trade compliance business operations include (1) establishing and 
maintaining trade accounts; (2) processing and releasing cargo for import or 
export; tracking and monitoring of conveyances, cargo, and individuals 
involved in importing and exporting; (3) liquidating import entries including the 
processing of protests and decisions and the potential authorization of 
refunds; and (4) ruling on import and export legal, regulatory, and policy 
issues. 



12 



Page 21 



GAO-04-719 Customs Modernization 



Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 



A £^ Background 



Accountability * Integrity * Reliability 



Description of ACE Technical Architecture 



The ACE technical architecture is to consist of layers or tiers of computer 
technology: 

• The Client Tier includes user workstations and external system interfaces. 

• The Web & Interface Tier provides the mechanisms for the user workstations 
and external systems to access ACE. 

• The Enterprise Application Integration Tier provides the middleware for 
integrating and routing information between ACE software applications and 
legacy systems. 

• The Applications Tier includes software applications comprising commercial 
products (e.g., SAP 1 ), custom-developed software, and existing legacy 
systems that provide the applications supporting business processes. 

• The Data Tier includes the business data, storage resources, and access 
services used by the ACE software applications and the legacy systems. 

Security and data privacy are embedded in all five layers. 

1 SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate 
but integrated business functions. CBP is using SAP as part of a joint project between its Office of Finance to support 
financial and property management and the CBP Modernization Office (CBPMO) to support such ACE functions as customer 
relationship management. 
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ACE Technical Architecture 



Accountability * Integrity * Reliability 



Simplified view of ACE technical architecture 




Security 



Source: GAO based on CBP data. 
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Background 
Acquisition Strategy 



CBP's initial strategy provided for the acquisition of ACE in four increments over 4 
years. In September 2002, CBP modified this strategy to acquire and deploy the 
first two increments in two releases, with all increments and releases acquired over 
4 years. In October 2003, CBP changed plans to acquire ACE through the 
acquisition and deployment of 10 releases over 6 years. 

The following figure maps the initial four increments to the 10 releases now 
planned, 7 of which are called "releases" (e.g., Release 1, Release 2, ... Release 
7) and 3 of which are called "selectivity" (e.g., Selectivity 1 , Selectivity 2, and 
Selectivity 3). 
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Background 
Acquisition Strategy 



Evolution of ACE from 4 increments to 10 releases 



Increment 1 



Increment 3 



Increment 4 



►| Increment 1 , release 1 



-► Release 1 



Release 2 



Increment 1 , release 2 



■► Release 3 



Release 4 



Increment 2 




Increment 2, release 1 


► 


Release 5 



Increment 2, release 2 



► Increment 3, release 



Selectivity 1 
Release 6 



Selectivity 2 



Increment 3, release 2 
-►I Increment 4 



X- H Release 7 



Selectivity 3 



Source: GAO based on CBP data. 
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Composite View of ACE Releases 



The functionality associated with, status of, and plans for the 10 ACE releases are 
as follows. 

Release 1: Provide IT infrastructure — computer hardware and system software — to 
simultaneously support operating the subsequent system releases. This release 
was deployed in October 2003 and is operating. 

Release 2: Provide initial group of CBP national account managers 1 and 41 
importers access to account information, such as trade activity. This release was 
deployed in October 2003 and is operating. 

Release 3: Provide additional account managers and importers, as well as brokers 
and carriers, 2 access to account information; provide initial financial transaction 
processing and CBP revenue collection capability, allowing importers and their 
brokers to make monthly payments of duties and fees. This release is under 
development and scheduled for deployment in August 2004. 



1 CBP national account managers work with the largest importers to ensure their compliance with trade laws. 

2 Brokers obtain licenses from CBP to conduct business on behalf of the importers by filling out paperwork and obtaining a 
bond; carriers are individuals or organizations engaged in transporting goods for hire. 
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Background 



Accountability * Integrity * Reliability 



Composite View of ACE Releases 



Release 4: Provide truck manifest 1 processing and interfacing to legacy 
enforcement systems and transaction history data. This release is under 
development and scheduled for deployment in February 2005. 

Release 5: Provide legacy systems interface for airport, seaport, and rail crossing 
inspectors for accessing supply chain, enforcement, and compliance data, as well 
as additional manifest functionality for selected federal government agencies 
(including the Food and Drug Administration) and the trade community. This 
release is scheduled for deployment in August 2005. 

Selectivity 1 : Identify shipments that pose a security risk using new and different 
combinations of data than current legacy systems permit, as well as end-user 
defined threat identification criteria. This release is scheduled for deployment in 
September 2005. 



1 Manifests are lists of passengers or invoices of cargo for a vehicle, such as a truck, ship, or plane. 
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Composite View of ACE Releases 



Release 6: Provide additional account management functionality for daily 
statement creation, refund identification and generation, and expanded payment 
acceptance; license, permit, and certificate granting, tracking, and revoking; and 
cargo, conveyance, and equipment tracking. This release is scheduled for 
deployment in December 2006. 

Selectivity 2: Identify shipments that pose a security risk during authorized 
movements. This release is scheduled for deployment in May 2006. 

Release 7: Provide manifest processing for cargo moving between modes of 
transportation; additional cargo and conveyance tracking and initial export 
processing functions; importer activity summary; and regulatory audit, protest, and 
drawback 1 processing. This release is scheduled for deployment in June 2007. 

Selectivity 3: Identify imports and exports (e.g., mail, courier, hand-carried items, 
and shipments transitioning between modes of transportation) that pose a security 
risk. This release is scheduled for deployment in November 2006. 



1 A refund of duties on an imported product subsequently exported or used to produce a product for export. 
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Background 

Composite View of ACE Releases 



In addition to the expanding functionality planned from one release to the next, the 
number of users and locations is planned to grow with each release, as shown 
below. 



Number of Users 



Number of Ports 



800000 
700000 
600000 
500000 
400000 
300000 
200000 
100000 






1 &2 3 



1 &2 3 



Release 



Release 



Note: Selectivity 1 , 2, and 3 are planned to be accessible by 37,000 or more users via the CBP network and are not included in the above figures. 
Source: CBP. 
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Background 
Contract Overview 



CBP is acquiring and implementing ACE through an indefinite-delivery, indefinite- 
quantity contract 1 awarded on April 27, 2001, to IBM Global Services. IBM and its 
subcontractors are collectively called the e-Customs Partnership (eCP). Through a 
series of contract task orders, CBP plans to acquire and implement ACE. 

The following table describes and provides the status of the executed eCP task 
orders. 




GAP 

Accountability * Integrity * Reliability 



1 An indefinite-delivery, indefinite-quantity contract provides for an indefinite quantity, within stated limits, of supplies or 
services during a fixed period of time. The government schedules deliveries or performance by placing orders with the 
contractor. 
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Status and description of eCP task orders 



No. 


Name 


Start 


Status 


Description 


001 


Program 
management 


August 2001 


Completed 
July 2003 


Initial program and project management; continued by 
task 009. 


002 


Enterprise 
architecture and 
engineering 


August 2001 


Completed 
June 2003 


Initial enterprise architecture and system engineering; 
continued by task 010. 


003 


Requirements 
and planning 


August 2001 


Completed 
July 2002 


Initial requirements development and program planning 
effort; continued by tasks for specific 
increments/releases. 


004 


Releases 1 and 2 


February 2002 


Completed 
October 2003 


Design, development, testing, and deployment of 
Releases 1 and 2 (initially intended to build Increment 1, 
which was subsequently divided into four releases; 
Releases 3 and 4 are covered by task 008) 


005 


Requirements 
definition 


February 2002 


Ongoing 


Development of Release 5 project plan, documentation 
of ACE business processes, and development of an 
ACE implementation strategy. 


006 


Enterprise 

process 

improvement 


February 2002 


Completed 
December 2003 


Enterprise process improvement integration. 



Source: GAO analysis based on CBP data. 
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Status and description of eCP task orders (cont'd) 



No. 


Name 


Start 


Status 


Description 


007 


International 
Trade Data 
System 


January 2002 


Ongoing 


Assistance for participating government agencies to define 
requirements for an integrated ACE/ITDS system. 


008 


Releases 3 and 4 


August 2002 


Ongoing 


Design, development, testing, and deployment of Releases 3 
and 4. 


009 


Foundation 

program 

management 


February 2003 


Completed 
October 2003 


Follow-on to task 001 to continue program and project 
management activities; continued by Task 016. 


010 


Foundation 
architecture and 
engineering 


February 2003 


Completed 
December 2003 


Follow-on to task 002 to continue enterprise architecture and 
system engineering activities; continued by task 017. 


011 


Infrastructure and 
facilities 


August 2002 


Completed 
March 2003 


Acquisition and setup of the necessary infrastructure and 
facilities for the contractor to design, develop, and test 
releases. Infrastructure and facilities work is now part of task 
009. 


012 


Operations and 
maintenance 


April 2003 


Ongoing 


Establishment of the infrastructure to operate and maintain 
releases. 



Source: GAO analysis based on CBP data. 



k GAO 



23 



Page 32 



GAO-04-719 Customs Modernization 



Appendix I 

IT: Early Releases of Customs Trade System 
Operating, but Pattern of Cost and Schedule 
Problems Needs to Be Addressed 



i 


GAO 

■ Accountability * Integrity * Reliability 




Background 
Contract Tasks 


Status and description of eCP task orders (cont'd) 


No. 


Name 


Start 


Status 


Description 


013 


Legacy scripts 
modernization 


June 2003 


Completed 
November 2003 


Conversion of scripts for interfacing desktop PC 
applications (MS Word and Excel) and mainframe 
computer applications. 


014 


Knowledge- 
based risk 
management 


September 2003 


March 2004 

planned 

completion 


Development, demonstration, and delivery of a 
prototype to provide CBP insight into whether 
knowledge-based risk management should be used in 
ACE. 


015 


Technology 
prototypes 


October 2003 


July 2004 

planned 

completion 


Development and demonstration of technology 
prototypes to provide CBP insight into whether the 
technologies should be used in ACE. 


016 


Foundation 
program 
management - 
workforce 
transformation 


February 2004 


Ongoing 


Program management and support to organizational 
change management through activities such as impact 
assessments, end user training, communication, and 
outreach. 


017 


Business 
analysis 


January 2004 


Ongoing 


Planning for integrated business framework and 
evaluation of technology options for future releases. 


Source: GAO analysis based on CBP data. 
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Background 
ACE Schedule 




(3 Product Definition Completion Review 
**C? Preliminary Design Review 
□ Critical Design Review 
A Test Readiness Review 

Production Readiness Review 
# Operational Readiness Review 

Source: GAO based on CBP data. 
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Chronology of ACE Expenditure Plans 



Since March 2001 , five ACE expenditure plans have been submitted. 1 

• On March 26, 2001 , CBP submitted to its appropriations committees the first 
expenditure plan seeking $45 million for the modernization contract to sustain 
CBPMO operations, including contractor support. The appropriations 
committees subsequently approved the use of $45 million, bringing the total 
ACE funding to $50 million. 

• On February 1 , 2002, the second expenditure plan sought $206.9 million to 
sustain CBPMO operations; define, design, develop, and deploy Increment 1 , 
Release 1 (now Releases 1 and 2); and identify requirements for Increment 2 
(now Release 5 and Selectivity 1 in addition to part of Release 6 and 
Selectivity 2). The appropriations committees subsequently approved the use 
of $188.6 million, bringing total ACE funding to $238.6 million. 



1 1n March 2001 , appropriations committees approved the use of $5 million in stopgap funding to fund program management 
office operations. 
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Chronology of Five ACE Expenditure Plans 



On May 24, 2002, the third expenditure plan sought $190.2 million to define, 
design, develop, and implement Increment 1 , Release 2 (now Releases 3 and 
4). The appropriations committees subsequently approved the use of $190.2 
million, bringing the total ACE funding to $428.8 million. 

On November 22, 2002, the fourth expenditure plan sought $314 million to 
operate and maintain Increment 1 (now Releases 1, 2, 3, and 4), design and 
develop Increment 2, Release 1 (now Release 5 and Selectivity 1), and to 
define requirements and plan Increment 3 (now part of Releases 6 and 7 and 
Selectivity 2 and 3). The appropriations committees subsequently approved 
the use of $306.4 million, bringing total ACE funding to $735.2 million. 

On January 21 , 2004, CBP submitted its fifth expenditure plan, seeking 
$318.7 million for ACE implementation infrastructure and support, ACE 
operations and maintenance, and definition and design of Release 6 and 
Selectivity 2. 
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Mapping of ACE Releases to Expenditure Plans 



How expenditure plan 1 requests have funded or are to fund life cycle phases of 
each ACE release 

Release Expenditure Plan 2 Expenditure Plan 3 Expenditure Plan 4 Expenditure Plan 5 



Release 1 




□□□□□ 






Release 2 




□□□□□ 






Release 3 


□□□□□ 


■■■■■ 






Release 4 


□□□□□ 


■■■■■ 




□□□□□ 


Release 5 




□□□□□ 




□□□□□ 


Selectivity 1 




□□□□□ 




□□□□□ 


Release 6 




□□□□□ 






Selectivity 2 


□□□□□ 


□□□□□ 


□□□□□ 




Release 7 


□□□□□ 


□□□□□ 




□□□□□ 


Selectivity 3 


□□□□□ 


□□□□□ 


□□□□□ 


□□□□□ 



I D D D D Requirements development 

□ Design 

D D I D D Development 

□ Deployment 

D D D D I Operations and maintenance 

Source: GAO analysis based on CBP data. 

1 Expenditure Plan 1 funded program management and planning only. 
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Summary of Expenditure Plan Funding 



Summary of the ACE fiscal year 2004 expenditure plan 



Plan activity 

Release 6 definition and design 
Selectivity 2 definition and design 
Implementation infrastructure and support 
Foundation program management 
Foundation architecture and engineering 
Workforce transformation and training 
Operations and maintenance 
CBPMO costs 

Federally Funded Research and Development Center 

International Trade Data System 

SAP b 

Management reserve 

Total Expenditure Plan Funding 3 $318.7 

Source: GAO based on CBP data. 
a Millions of dollars. 

b SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate 
but integrated business functions. CBP is using SAP as part of a joint project between its Office of Finance to support 
financial and property management and the CBPMO to support such ACE functions as customer relationship management. 
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Objective 1 Results 
Legislative Conditions 



The fiscal year 2004 ACE expenditure plan satisfied or partially satisfied each of 
the legislative conditions. 

Condition 1. The plan, in conjunction with related program documentation and 
program officials' statements, satisfied the capital planning and investment control 
review requirements established by OMB, including Circular A-1 1 , part 7, which 
establishes policy for planning, budgeting, acquisition, and management of federal 
capital assets. 

Details of our analysis are shown on the table that follows. 
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Objective 1 Results 
Legislative Condition 1 



Examples of A-1 1 conditions Results of our analysis 

1 . Indicate whether the project was The plan was approved by the DHS IRB, which is 
approved by the IRB and chaired by the Deputy Secretary and composed of 
reviewed by Chief Financial and department executives, including DHS's Chief Financial 
Procurement Officers and Procurement Officers. 

2. Provide justification and The plan provides a high-level justification for ACE. 
describe acquisition strategy Supporting documentation describes the acquisition 

strategy for all ACE releases, including Release 6 and 
Selectivity 2 activities identified in the plan. 

CBP's November 2003 cost/benefit analysis 
summarizes current life cycle and return on investment 
cost information. CBP reports that the overall ACE 
benefit to cost ratio for CBP alone is 1 .43 a and 6.66 b for 
CBP, other government agencies, and the trade 
community. 

a The government benefit to cost ratio of 1 .43, which is for CBP only, was calculated by dividing the present value benefits of 
$3.97 billion by the present value cost of $2.78 billion. According to the cost/benefit analysis, benefits are generated from the 
efficiency gains of ACE and the decommissioning of the legacy Automated Commercial System. 

b The national benefit to cost ratio of 6.66, which is for CBP, other government agencies, and the trade community, was 
calculated by dividing the present value benefits of $18.55 billion by the present value cost of $2.79 billion. The ratio may not 
be exact due to rounding. 
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3. Summarize life cycle costs and 
cost/benefit analysis, including 
the return on investment 
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Objective 1 Results 
Legislative Condition 1 


Examples of A-11 conditions 


Results of our analysis 


4. Address enterprise 
architecture 


Alignment of ACE to the enterprise architecture has yet to be 
determined, but DHS and CBP officials stated that they plan to 
address this issue. 



5. Address security and privacy The security of Release 2 was certified on August 22, 2003, 

and accredited on September 16, 2003; CBP plans to certify 
and accredit future releases. CBP is in the process of preparing 
a privacy impact assessment for Release 3 and expects to 
complete it by March 2004. 



6. Address Section 508 Release 2 was deployed without requiring Section 508 

compliance 01 compliance by the system development contractor. The 

requirement for Section 508 compliance was not recognized 
and was not reflected in the designs of Releases 2 and 3. CBP 
decided to partially address Section 508 compliance in Release 
3 and defer complete compliance to future releases. CBP plans 
to assess Release 3 with respect to Section 508 compliance, 
and implement an action plan to ensure that noncompliant 
areas are corrected in future releases. 

Source: GAO. 

c For more details, see the results of our analysis for the second legislative condition. 

"Section 508 of the Rehabilitation Act (29 U.S.C. 794d), as amended by the Workforce Investment Act of 1998 (P.L. 105-220), 
August 7, 1998, requires federal agencies to develop, procure, maintain, and use electronic information technology in a way that 
ensures that the technology is accessible to people with disabilities. 
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Objective 1 Results 
Legislative Condition 2 



Condition 2. The plan and related program officials' statements provide for 
complying with the DHS enterprise architecture. 

DHS released version 1 of the architecture in October 2003, 1 and plans to issue 
version 2 in September 2004. 

According to the DHS Chief Information Officer (CIO), DHS is developing a process 
to align its systems modernization efforts, such as ACE, to its enterprise 
architecture. Alignment of ACE to the enterprise architecture has yet to be 
determined, but DHS CIO and CBP officials stated that they plan to address this 
issue. Moreover, the fiscal year 2004 expenditure plan states that Release 6 and 
Selectivity 2 will be aligned with the DHS enterprise architecture. 
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1 Department of Homeland Security Enterprise Architecture Compendium Version 1.0 and Transitional Strategy. 
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Objective 1 Results 
Legislative Condition 3 



Condition 3. The plan, in conjunction with related program documentation, partially 
satisfies the condition of compliance with the acquisition rules, requirements, 
guidelines, and systems acquisition management practices of the federal 
government. 

The Software Acquisition Capability Maturity Model (SA-CMM), developed by 
Carnegie Mellon University's Software Engineering Institute (SEI), is consistent 
with the acquisition guidelines and systems acquisition management practices of 
the federal government, and it provides a management framework that defines 
processes for acquisition planning, solicitation, requirements development and 
management, project management, contract tracking and oversight, and 
evaluation. 

On November 5, 2003, SEI assessed ACE acquisition management against the 
SA-CMM and assigned a level 2 rating, indicating that CBPMO has instituted basic 
acquisition management processes and controls in the following areas: acquisition 
planning, solicitation, requirements development and management, project 
management, contract tracking and oversight, and evaluation. 
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Objective 1 Results 
Legislative Condition 3 



In June 2003, the Department of the Treasury's Office of Inspector General (OIG) 
issued a report on the ACE program's contract, concluding that the former Customs 
Service, now CBP, did not fully comply with Federal Acquisition Regulation 
requirements in the solicitation and award of its contract because the ACE contract 
is a multiyear contract and not an indefinite delivery /indefinite quantity (IDIQ) 
contract. Further, the Treasury OIG found that the ACE contract type, which it 
determined to be a multiyear contract, is not compatible with the program's stated 
needs for a contract that can be extended to a total of 15 years because multiyear 
contracts are limited to 5 years. Additionally, the Treasury OIG found that Customs 
combined multiyear contracting with IDIQ contracting practices. For example, it 
plans to use contract options to extend the initial 5-year performance period. 

CBP disagrees with the Treasury OIG conclusion. 

To resolve the disagreement, DHS asked GAO to render a formal decision. We are 
currently reviewing the matter. 
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Objective 1 Results 
Legislative Condition 4 



Satisfies condition 4. The plan satisfies the condition that it be reviewed and 
approved by the CBP IRB, DHS, and OMB. 

On July 15, 2003, the CBP IRB approved the plan. DHS subsequently approved 
the plan on November 20, 2003, and OMB approved the plan on January 20, 2004. 
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Objective 1 Results 
Legislative Condition 5 

Satisfies condition 5. The plan satisfies the condition that it be reviewed by GAO. 
Our review was completed on March 4, 2004. 
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Objective 2 Results 
Open Recommendation 1 



Open recommendation 1: Before building each ACE release (i.e., beginning 
detailed design and development), certify to Customs' House and Senate 
appropriations subcommittees that the enterprise architecture has been sufficiently 
extended to provide the requisite enterprise design content and has been updated 
to ensure consistency and integration across business areas. 

Status: Complete. 

In response to our recommendation made in March 2002, the U.S. Customs 
Service established a process for certifying to its appropriations committees that 
the Customs enterprise architecture had been sufficiently extended and updated 
before detailed design and development of each ACE release was begun. 

In March 2003, the U.S. Customs Service business operations supported by ACE 
became part of the DHS CBP. 
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Objective 2 Results 
Open Recommendation 1 



In May 2003, CBP used the process developed by the former Customs Service to 
certify Releases 3 and 4 against the former Customs enterprise architecture. At this 
time, DHS also committed to developing a single, departmentwide enterprise 
architecture to replace DHS component agency/bureau architectures. DHS 
released version 1 of its enterprise architecture in October 2003, 1 and plans to 
issue version 2 of its architecture in September 2004. 

According to the DHS CIO, DHS is developing a process to ensure alignment of its 
systems modernization efforts, including ACE, to its enterprise architecture. While 
this alignment step for ACE has not occurred yet, the fiscal year 2004 expenditure 
plan states that Release 6 and Selectivity 2 will be aligned with the DHS 
architecture. 
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1 Department of Homeland Security Enterprise Architecture Compendium Version 1.0 and Transitional Strategy. 
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Objective 2 Results 
Open Recommendation 2 



Open recommendation 2: Develop and implement a rigorous and analytically 
verifiable cost estimating program that embodies the tenets of effective estimating 
as defined in SEI's institutional and project-specific estimating models. 1 

Status: In progress. 

The program office has developed and begun to implement a cost estimating 
program that embodies SEI's models. For example, CBP hired a contractor to 
define and implement a cost estimating program. The contractor has responsibility 
for development of ACE life cycle cost estimates and independent cost estimates to 
support budget planning, budget execution, and contract task order evaluation. 
Additionally, the program office has defined and documented processes for 
estimating expenditure plan costs and management reserve in addition to 
preparing independent cost estimates. Further, CBP tasked its support and 
modernization contractors with ensuring that the expenditure plan estimates are 
evaluated against SEI criteria for validating software cost and schedule estimates. 

1 SEI's institutional and project-specific estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost 
and Schedule Estimating Capabilities of Software Organizations and A Manager's Checklist for Validating Software Cost and 
Schedule Estimates, respectively. 
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Objective 2 Results 
Open Recommendation 3 



Open recommendation 3: Immediately develop and implement a human capital 
management strategy that provides both near- and long-term solutions to program 
office human capital capacity limitations, and report quarterly to the appropriations 
committees on the progress of efforts to do so. These efforts should include 
defining the office's skill and capability needs in terms that will allow the program 
office to attract qualified individuals and that will provide sufficient rewards and 
training, linked to performance, to promote their retention. 

Status: In progress 

The program office developed and began implementing a human capital 
management plan. This plan called for addressing several areas, including 
identification of required core competencies (i.e., defined knowledge and skill 
requirements), assessing existing capabilities, developing a training program, and 
filling vacant positions. 
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Objective 2 Results 
Open Recommendation 3 



In its June and October 2003 reports to its appropriations committees on progress 
in addressing these areas, the program office indicated that it had defined core 
competencies, was assessing capabilities and developing an annual training plan, 
and had filled three vacancies and developed a plan for filling its remaining 
vacancies. However, the program office has continued to experience difficulty in 
filling key positions, including three responsible for organizational change 
associated with adopting the business process changes embedded in ACE 
functionality. 

Currently, program officials told us that the capability assessment and the annual 
training plan were completed and both are awaiting approval. They also told us that 
they have suspended development of the original plan for filling vacancies, and in 
January 2004 began implementing a new staffing plan that was developed in 
response to a DHS IRB statement that the program office has insufficient 
government program management staff. We made this same observation, along 
with recommendations to address it, 2 years ago. 
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Objective 2 Results 
Open Recommendation 4 



Open recommendation 4: Develop and implement process controls for SEI's SA- 
CMM level 2 key process areas and for the level 3 acquisition risk management 
key process area; develop and implement each of the missing SA-CMM key 
practices for these key process areas, and until this is accomplished, report to the 
appropriations committees quarterly on the progress of efforts to do so. 

Status: Complete 

CBP has taken steps to develop and implement each of the key process areas for 
the SA-CMM level 2 key process areas and the level 3 acquisition risk 
management area. SEI assessed the acquisition management of Releases 1 
through 4 against the SA-CMM level 2 criteria. On November 5, 2003, SEI 
assigned the program a level 2 rating, meaning that the program has established 
basic acquisition management processes. 
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Objective 2 Results 
Open Recommendation 4 



SEI also conducted an unrated assessment of the program's acquisition risk 
management capability in May 2003. The assessment identified two weaknesses: 
(1) risk management was not integrated into acquisition planning and (2) a defined 
organizational process was not used to produce the program's acquisition risk 
management plan. CBP has addressed these weaknesses by updating its 
acquisition planning process to include risk management activities and by 
documenting an acquisition risk management process. 

CBP's October 2003 report to its appropriations committees reported on the status 
of these acquisition management improvement efforts. 
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Open recommendation 5: Establish an independent verification and validation 
(IV&V) function to assist CBP in overseeing contractor efforts, such as testing. 

Status: In progress 

The program office has established an integrated independent verification and 
validation (l 2 V 2 ) function, and it has designated its l 2 V 2 contractor. 

According to CBP's l 2 V 2 strategy, the contractor is to review key eCP deliverables 
and products and provide assessments and reports at various life-cycle stages and 
milestones. Further, the strategy recognizes the importance of maintaining 
independence in the conduct of l 2 V 2 and outlines (1) a reporting structure that 
allows the contractor to report its findings to CBP executives and (2) an approach 
to assigning staff to l 2 V 2 tasks that is to ensure contractor independence from 
program management pressures. 
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Objective 2 Results 
Open Recommendation 5 



However, other program documentation, such as the fiscal year 2004 expenditure 
plan, the contractor's statement of objectives, and the modernization program work 
breakdown structure, describes roles for the l 2 V 2 contractor that are so integral to 
ACE development, testing, and deployment, as well as ACE program management, 
that they raise questions about how the contractor could also perform independent 
ACE-related assessments. For example, they assign the l 2 V 2 contractor 
responsibility for 

• supporting development of long-range and short-term strategic plans for the 
modernization program; 

• developing the task orders that contractually specify eCP's work; and 

• developing incentives for improving eCP's performance. 

CBP officials stated that they recognize the importance of ensuring the 
independence of their verification and validation function and plan to address this 
situation. 
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Open Recommendation 6 



Open recommendation 6: Have future ACE expenditure plans specifically 
address any proposals or plans, whether tentative or approved, for extending and 
using ACE infrastructure to support other homeland security applications, including 
any impact on ACE of such proposals and plans. 

Status: Planned 

Program officials acknowledge the potential for ACE infrastructure to support other 
DHS system applications and the potential for integrating ACE data and 
applications with those systems. To begin preparing for this potential, program 
officials told us they have focused initially on discussing collaboration opportunities 
with DHS's United States Visitor and Immigrant Status Indicator Technology (US- 
VISIT) system. 1 To date, those discussions have included 

• sharing lessons learned on leveraging contractors and 

• coordinating enterprise architecture activities. 
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1 US-VISIT is a govern mentwide program to collect, maintain, and share information on foreign nationals for enhancing 
national security and facilitating legitimate trade and travel, while adhering to U.S. privacy laws and polices. 
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Objective 3 Results 
Observation 1 



Observation 1 : Release 1 and 2 testing revealed a sufficient volume and 
significance of system defects to impact schedule commitments. 

Development of each ACE release includes software integration, system 
integration, and system acceptance testing by eCP, followed by user acceptance 
testing by CBP. Generally, the purpose of these tests is to identify defects or 
problems in either meeting defined system requirements or in satisfying system 
user needs. A brief description of the four types of testing is provided below. 
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Test 


Description 


Software integration test 


Verify that the software subsystems and modules perform as 


(SWIT) 


intended. 


System integration test 


Verify that related system, subsystem, or module components 


(SIT) 


are capable of integrating and interfacing with each other. 


System acceptance test 


Verify that the developed system, subsystem, or module 


(SAT) 


operates in accordance with requirements. 


User acceptance test 


Verify that the functional scope of the release meets the 


(UAT) 


business functions for the users. 



Source: eCP. 
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Release 1 user acceptance testing revealed problems. 

According to CBP's test results report, user acceptance testing (UAT) for Release 1 
was intended to ensure that the design of the users' workstation screens, the 
capabilities for navigating among screens, and the functionality associated with 
these screens satisfied user needs. 1 Additionally, CBP's stated goal of Release 1 
UAT was to collect feedback from users and identify defects and improvement 
opportunities. 

UAT began on January 8, 2003, and was completed on January 24, 2003. It 
consisted of users (1) completing 28 test cases and (2) responding to 10 questions 
designed to solicit their satisfaction (on a scale of 1 to 5, with 1 being very poor and 
5 being excellent) about different aspects of Release 1, such as screen design, 
quality, reliability, availability, response time, and training. 
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1 1n GAO-03-406, we reported that the first ACE release had completed SAT. 
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Objective 3 Results 
Observation 1 



UAT results show that the average user satisfaction rating for Release 1 was 
slightly less than 3 out of 5, and that this resulted in 38 defects, referred to as 
program trouble reports (PTRs). Of these 38, 1 1 were judged to be either critical or 
severe. These and other categories of PTRs are described below. 

Category Description 

Critical Defect prevents or precludes the performance of an operational or mission-essential 

capability, jeopardizes safety or security, or causes the system, application, process, or 
function to fail to respond or end abnormally. 

Severe Defect prevents or precludes system from working as specified or produces an error that 

degrades or impacts system performance or user functionality. 

Moderate Defect prevents or precludes system from working as specified or produces an error that 
degrades or impacts system performance or user functionality. An acceptable (reasonable 
and effective) workaround is in place that rectifies the defect until a permanent fix can be 
made. 

Minor Defect is inconsequential, cosmetic, or inconvenient but does not prevent users from using the 

system to accomplish their tasks. 

Source: GAO based on eCP data. 

In order to successfully complete UAT and make Release 1 operational, CBP 
reported that it resolved 17 PTRs and deferred to Release 2 the remaining 21 . 
About 3 months after UAT concluded, CBP reported that all 38 PTRs were 
resolved. 
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Objective 3 Results 
Observation 1 



Release 2 integration, acceptance, and user acceptance testing revealed problems 
that required additional time to address. 

System integration testing (SIT). As previously stated, SIT is intended to verify that 
related system, subsystem, or module components are capable of integrating and 
interfacing with each other. Successfully completing SIT requires that all test cases 
achieve a "test passed" status, meaning that the test procedures were successfully 
executed and completed, and the test objectives were met. 

Release 2 SIT, which began on December 31 , 2002, and was to end on April 4, 
2003, consisted of 46 test cases. Of these 46, 33 were reported to be successfully 
completed. According to the test results, of the remaining 13 test cases, either they 
could not be executed for various reasons (such as a capability not being available 
in the current release) or a test case was changed during attempts to execute it. 
The 46 test cases, according to eCP's projection, were expected to disclose 389 
defects; however, they resulted in 745. 
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Objective 3 Results 
Observation 1 



When SIT concluded in June 2003 (about 2.5 months later than planned), 49 PTRs 
remained open (6 severe, 36 moderate, and 7 minor). However, because none of 
the tested items failed the SIT and the defects identified were either resolved or 
deferred to SAT or a later release, the SIT test procedures were judged to be 
successfully executed. 

As of January 2004, the program office reports that all the critical and severe PTRs 
have been resolved, and that 16 PTRs are still open, of which 1 1 are moderate and 
5 are minor. 
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Objective 3 Results 
Observation 1 



System acceptance testing (SAT). According to CBP's test results report, SAT is 
intended to show that a system performs according to its specified functional and 
performance requirements. To pass SAT, all test cases must pass without 
identification of critical or severe defects. 

Release 2 SAT was to begin April 7, 2003, and conclude on April 28, 2003, but 
because of the large number of unexpected defects, SAT began on May 1 , 2003, 
and ended on June 5, 2003, which is about a 1 month delay. Of the SAT test cases 
executed, all but 19 reportedly passed. Of these 19, one resulted in a critical PTR, 
which was closed within 2 days, and 18 produced moderate or minor PTRs. The 
program office reports that those 18 were either corrected or deferred to future 
releases based on implementation of acceptable procedural workarounds. 

As of January 2004, 1 1 of the deferred PTRs reportedly remain open. An example 
of a deferred PTR is Release 2's inability to identify duplicate accounts. According 
to a system integration manager, this defect was not categorized as critical or 
severe because an acceptable workaround was available (i.e., establishing a 
manual process to identify possible duplicate accounts until Release 3 is ready). 
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Objective 3 Results 
Observation 1 



User acceptance testing (UAT). According to CBP's test results report, UAT is 
intended to verify that the system meets the needs of the users. 

Release 2 UAT began on July 10, 2003, and was to end on August 15, 2003. It 
consisted of CBP and the trade community users executing 19 and 23 test cases, 
respectively, and being asked to rate the release in several areas using a 1 to 5 
scale (with 1 indicating very dissatisfied and 5 indicating very satisfied). The test 
areas addressed the account management functionality included in Release 2, 
such as creating a new account, managing account details, and creating reports. 
The average user satisfaction rating was slightly more than 4 out of 5, which 
indicated user satisfaction, according to the test results report. 

UAT nevertheless resulted in the identification of 17 defects. As of January 2004, 
15 of the associated PTRs are reported to have been resolved, and the remaining 
2 are categorized as moderate. 

The shaded portion of the graph on the following page shows the number of PTR's 
open (i.e., unresolved) each week during Release 2 testing. 
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Observation 1 



Release 2 Unresolved Defects by Week 
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Objective 3 Results 
Observation 2 



Observation 2: Requirements planned for Releases 1 and 2 have been 
deferred. 

As previously described, each ACE release is expected to contain specific 
functionality and performance capabilities. These capabilities are defined in a 
system requirements document, which lists the overall requirements for the entire 
ACE system. The requirements specification set breaks these into more detailed 
requirements and assigns them to releases. 

Releases 1 and 2 were originally planned as a single release that was to satisfy 
465 requirements. Subsequently, 103 of these requirements were deferred to 
Release 3 and 4, and 14 requirements were deferred to later releases, for a total of 
117 deferred requirements. Also, 25 of 465 requirements were deleted from the 
baseline. In total, 142 requirements were either deferred or deleted from the 
original baseline, representing about 31 percent of the original baseline of 465 
requirements. 

At the same time, 36 new requirements were added to the baseline for Release 1 
and 2, meaning that Release 1 and 2 satisfied 359 total requirements. The next 
slide summarizes the requirements changes in Release 1 and 2. 
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Objective 3 Results 
Observation 2 
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Objective 3 Results 
Observation 2 



According to the requirements specification set for Release 3 and 4, the following 
categories of capabilities and functionality were deferred from Releases 1 and 2 to 
Releases 3 and 4: 

• document management, document retention, workflow, and data recovery; 

• network response time; and 

• security (remote access authentication, data security during network transfer, 
audit trails). 

According to program officials, the deferred security requirements must be 
implemented by Release 4. 

Program change requests cite the reasons for the respective requirements deferral. 
Reasons include the following: 

• Document management and retention requirements were deferred because 
neither Release 1 nor 2 needed to be able to accommodate either paper or 
electronic documents. 
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• Workflow requirements were deferred because eCP and CBP needed 
additional time to evaluate and select among commercial product options. 

• Data recovery functionality was deferred because CBP chose to delay 
implementation of disaster recovery for ACE until Releases 3 and 4. 

• Broker and carrier account functionality was deferred because full support for 
the functionality was not needed until Releases 3 and 4. 
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Objective 3 Results 
Observation 3 



Observation 3: Release 2 delays have set in motion a pattern of increased 
reliance on concurrent activities, continued release delays, and cost 
overruns. 

As we previously reported, 1 the importance of ACE to our country's security and 
trade facilitation prompted CBP to take steps aimed at expediting ACE acquisition 
and deployment, including introducing greater reliance on performing release 
design, development, and testing activities concurrently. While such concurrency 
can permit faster acquisition and deployment, it also introduces risks, which the 
ACE contractor has said include limited understanding of requirements before 
design and development activities begin, uncertainty regarding the timely 
availability of commercial hardware and software products, and increased near- 
term funding requirements. Other risks include contention for limited resources 
(such as key personnel, as well as development and testing equipment and 
facilities) and dependencies among releases not being met. 
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1 U.S. General Accounting Office, Customs Service Modernization: Management Improvements Needed on High-Risk 
Automated Commercial Environment Project GAO-02-545 (Washington, D.C.: May 13, 2002). 
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Objective 3 Results 
Observation 3 



Time required to resolve defects resulted in overlap of Release 2 integration and 
acceptance testing. 

Resolving Release 2 defects also resulted in more overlap between testing 
activities. The original Release 2 testing schedule projected an overlap of about 1 .5 
months between SWIT and SIT, and no overlap between SIT and SAT. However, 
due to the extension of testing periods caused by the high number of defects, the 
overlap between SWIT and SIT was about 4.5 months, and the SAT period was 
completely concurrent with SIT. The next slide shows the overlap of Release 2 
testing activities. 
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Release 2 Testing Periods 
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Delays in completing Release 2 increased its overlap with Release 3 and have 
contributed to Release 3 delays. 

According to program officials and variance reports, some ongoing Release 3 work 
had to be curtailed when key resources slated for Release 3 (e.g., test and 
development environment, equipment, and key personnel) had to be diverted to 
address Release 2 PTRs and complete Release 2 test activities. Although some 
resource diversion continued through June 2003, as Release 3 resources were 
used to complete Release 2 SIT (including regression testing of software changes 
made to address PTRs), one report states that after June 2003, about 50 percent 
of the planned Release 3 SIT resources were redirected to support Release 2 
deployment activities. 

According to program officials, Release 1 delays, coupled with Release 2 
dependencies on Release 1 and subsequent contention for key resources between 
Releases 2 and 3, have had a domino effect, causing Release 3 resource shortfalls 
and consequent delays. For example, as of January 2004, a delay of 5 to 6 months 
in the March 2004 milestone for determining Release 3 operational readiness was 
reported. 
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Release 3 delays will increase overlap with Release 4 and could cause Release 4 
delays. 

According to program officials and documentation, Release 3 and 4 development 
began in August 2001, concurrent with ongoing Release 1 and 2 testing, but before 
they were aware of the large number of Release 1 and 2 defects. According to 
these officials, work was undertaken concurrently to minimize the time between 
releases and thereby maximize the efficiency in deploying releases. 

While these officials agreed that the cascading Release 1 and 2 problems will 
continue to impact Release 3, they stated that Release 4 will not be affected. One 
reason that they cited is that Release 4 will be tested in a recently acquired 
separate laboratory environment from Release 3, thus eliminating the contention 
for these key resources. 

Despite these statements, program variance reports show that Release 4 is being 
impacted. According to the reports, Release 3 resources targeted for Release 4 are 
not becoming available as early as planned because of delayed Release 3 
development and SWIT activities. Unless resources from an alternative source are 
forthcoming, this could impact Release 4 schedule commitments. 
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Increased concurrency and schedule delays resulted in Releases 1 through 4 
costing more than planned. 

The baseline cost estimate for Releases 1 and 2 was $86.1 million. However, the 
actual cost was reported to be $109.4 million — an overrun of $23.3 million (27 
percent). 

The baseline cost estimate for Release 3 and 4 was $149.5 million. However, CBP 
now estimates the cost to be $185.7 million — an overrun of about $36.2 million (24 
percent). 

According to program documentation, including cost variance analysis reports, 
factors contributing to the cost overruns included 

• resources to eliminate Release 1 and 2 defects; 

• overtime to minimize Release 3 and 4 schedule delays; 

• unavailability of testing and development environments; 

• Release 3 delays caused by resources being used to support delayed 
Release 2 activities; 
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• added Release 3 and 4 scope as a result of requirements shift from Release 
2; and 

• Release 3 receipt of unresolved defects from Release 2. 

CBP reported that it funded the $23.3 million Release 1 and 2 overrun with a $7.9 
million forfeiture of eCP's incentive fees and a $15.4 million reallocation of funds 
from Release 5. CBP also reported that it is considering funding the $36.2 million 
Release 3 and 4 overrun out of its accumulated management reserve. 
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Observation 4: Releases 1 and 2 are largely operating according to adjusted 
requirements; defect density has stabilized and system is mostly up and 
running when needed. 

Two indicators of how well a system is operating are defect density (i.e., the 
volume and severity of unresolved problems) and system availability (i.e., the time 
the system is operating satisfactorily, expressed as a percentage of time that the 
system is required to be operational). 

Releases 1 and 2 began operating on October 17, 2003. According to CBP, these 
releases are providing support to 73 import accounts and 3300 users. Further, 
these accounts represent (1) about seven percent of the 1 100 accounts expected 
to be supported by the end of 2004 and (2) about 19 percent and 20 percent of 
annual import value and trade duties collected, respectively. 
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Release 1 and 2 defect density has stabilized, but nontrivial defects remain. 

As indicated by the shaded area on the following graph representing the period 
since Releases 1 and 2 began operating, the defect density associated with both 
releases has stabilized. In particular, CBP defect data show that as of October 17, 
2003, these releases had no severe or critical PTRs, and 44 moderate and 21 
minor PTRs open. As of January 2004, these data show improvement, with 1 
severe, 34 moderate, and 17 minor PTRs open, and documentation shows that the 
severe defect was resolved on February 3, 2004. 
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Notwithstanding the stabilization of PTRs represented by these data, 34 moderate 
PTRs remain open, and by definition these relate to instances where Releases 1 
and 2 do not work as specified or produce an error that degrades or impacts 
system performance or user functionality. The only reason that they are not viewed 
as severe is that an acceptable (reasonable and effective) workaround is in place 
as a temporary fix, but such workarounds generally produce efficiency losses. 

Release 1 and 2 mostly available when needed. 

ACE is required to be available 99.9 percent of the time (24 hours a day, 7 days a 
week). Based on available eCP data, Releases 1 and 2 have met this requirement 
on all but 4 days since October 17, 2003, when they began operating. These days, 
outages, and the respective availability levels are provided in the table. 



Date 


System availability 


Outage time 


December 22, 2003 


82.51 percent 


4 hours, 12 minutes 


January 12, 2003 


99.65 percent 


5 minutes 


January 23, 2003 


96.53 percent 


50 minutes 


January 26, 2003 


97.15 percent 


41 minutes 



Source: eCP data. 
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For each of these outages, eCP documents a root cause analysis. Some of the 
outages can be attributed to problems such as an expired secure socket layer 
certificate, full log file for servers, and loss of database connection between two 
subsystems. In addition to identifying the root cause, eCP reports on the mitigation 
actions taken to prevent a reoccurrence of the problem. For example, to address 
the problem with the log file, eCP has opened a PTR for automated monitoring of 
file systems. Until the PTR is resolved, eCP reported that it is performing manual 
monitoring of the log file systems daily. 
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Observation 5: Following Release 1 problems, steps were taken to avoid 
future problems, but success of actions is unclear. 

Effective project management, including directing a contractor to perform a 
particular task (e.g., developing and implementing a corrective action plan), 
requires, among other things, a clear statement of objectives, identification of 
necessary resources, and defined measures of progress and results. 

As a result of the Release 1 cost overruns and schedule delays, CBP directed eCP, 
in February 2003, to develop and implement a plan to improve program 
performance. According to the plan, which is dated April 2003, the objective was to 
demonstrate serious, tangible, and measurable actions to resolve core program 
issues and achieve fundamental program improvement. 

The plan identified eight problem areas that eCP was to correct: 

• Ineffective leadership 

• Unclear roles and responsibilities 

• Misaligned priorities 
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• Weak system engineering 

• Poor program planning 

• Inaccurate and inconsistent financial data 

• Unmet contract commitments 

• Unstable system performance 

However, the plan did not define the level of resources required to execute it, and it 
did not specify how progress and improvement in each of these areas would be 
determined, including any specific measures and metrics. 

In October 2003, program management review documents show that all the 
corrective actions were reported to be completed. In January 2004, CBP conducted 
an evaluation to assess eCP's performance. Although the resulting January report 
was not available in time to be included in our review, CBP officials told us that 
implementation of eCP's corrective action plan is complete. They provided the 
following examples of corrective actions that eCP has taken to address each of the 
eight problem areas (see following slides). 
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Problem areas CBP-provided examples of eCP actions to correct problems 



Ineffective 
leadership 



Increased the number and caliber of resources. For example, eCP has brought on 
board an Enterprise Resource Planning 1 expert, a chief technology officer, a systems 
architect, and a workforce transformation leader. 



Unclear roles and 
responsibilities 



Business process owners now manage functional areas through the system life 
cycle. In the prior operating model, the business process teams would only be 
responsible for planning and defining the requirements, while the engineering teams 
would be responsible for building, preparing, and delivering the support for accounts 
management. In the new operating model, the business process owner would be 
responsible for ensuring that accounts management in Release 1 and 2 is planned 
and executed throughout all phases of the systems life cycle. 

Misaligned priorities Assessed each release to identify the desired business results that are being 

addressed so that there is visibility into the benefits and capabilities that are being 
gained through ACE. 



Enterprise resource planning is an integrated software solution used to manage a company's resources. 
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Problem areas CBP-provided examples of eCP actions to correct problems 

Weak system Improved testing and configuration management. For example, the environment 

engineering engineering group is now using a repeatable environment build process, and the 

business process owners participate in the testing activities and provide an internal 
check outside the software development team. 

Improved quality of products as a result of using the Ascendant Life Cycle 
Methodology 1 and tailoring it to the ACE program. For example, the deliverables now 
require less rework than previously and are more aligned with CBP's expectations. 

Revised the program plan to include more detail on the content of the various 
releases, descriptions of the end-state architecture, descriptions of testing and 
development needs integrated with cost and schedule; and cost and schedule 
estimates based on historical performance. 

Inaccurate and Improved financial controls and financial reporting as a result of new tools and 

inconsistent automation, evident in the monthly cost performance reviews and program 

financial data management reviews. 

Unmet contract Improved quality of deliverables with higher standards of acceptance evident with the 

commitments requirements, integrated baseline reviews, and preparation for proposals. 

Improved collaboration between eCP and the government with active monitoring and 
control. 



1 The Ascendant Life Cycle methodology is a specific development methodology for use with the SAP Enterprise Resource 
Planning software package. 
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Problem areas CBP-provided examples of eCP actions to correct problems 

Unstable system Stable and controlled testing environments. For example, for Release 3, more 
performance defects were identified in early testing activities, while in Release 2, the majority were 

identified in later tests, which is an indication of infrastructure stability. 
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Despite the examples offered, program officials could not point to metrics that 
demonstrate improved performance. Instead they stated that the corrective actions 
have not been implemented long enough to see substantial improvements and that 
the corrective actions will require time to demonstrate results. They added, 
however, that they have continued to see some improvement in the quality of 
deliverables. 

More concrete measures of performance that are available do not suggest 
adequate improvements. For example, as previously noted, Releases 3 and 4 have 
experienced 5 to 6 month delays and are over budget by $36.2 million. Program 
officials attributed the problems that occurred to problems experienced in Release 
1 affecting Releases 2, 3, and 4, before the corrective action plan was developed 
and implemented. They also stated that contract task orders are generally 
completed on time and under budget. 

According to program officials, the effect of the corrective actions will only be 
evident when eCP begins achieving milestones on schedule and within budget. 
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CBP expects these results to be seen with the delivery and approval of the Global 
Business Blueprint 1 in May 2004, which is the first major activity to be initiated after 
the corrective actions were implemented. 

Cost to develop and implement corrective action plan is unknown. 

Effective project management, including directing a contractor to perform a 
particular task (e.g., developing and implementing a corrective action plan), 
includes estimating the associated cost and monitoring the activity to ensure that 
cost estimates are met. 

According to program officials, they do not know the costs that the government 
incurred and the payments that the government made to eCP to develop and 
implement its plan for correcting problems associated with its delivery of proposed 
system capabilities on time and within budget. Rather, they said that these costs 
are spread across multiple contract tasks and are not identifiable. 

Program officials told us that while CBP requested eCP to track the costs of 
implementing the corrective actions, eCP representatives told them that it would be 
impossible to track these costs separately because they would be incurred as part 
of other ongoing tasks that eCP is required to perform under executed task orders. 

1 Global Business Blueprint is a planning project for future releases that is to establish an integrated business framework 

and identify appropriate technology enablers to support ACE cargo management. 
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Observation 6: Initial releases represent a small fraction of the total ACE 
software. 

A system's software size can be expressed in function points. Function points are 
standard units of measure that describe software in terms of end-user inputs and 
resulting outputs. They can be used to measure the size of any software regardless 
of the language, development method, or platform. Function point counts are 
derived from a variety of sources, including requirements documentation, 
interviews with end users, and business process descriptions. The International 
Function Point Users Group (IFPUG) maintains the standards for identifying and 
quantifying function points. The ACE system has been sized using function points 
with IFPUG-certified counters. 

All ACE releases collectively consist of 1 1 ,936 function points. Releases 1 and 2 
together contain 618 function points, and Releases 3 and 4 together contain 1 ,016 
function points. The remaining 10,302 function points are contained in the three 
selectivity releases and Releases 5, 6, and 7, with the majority (8400) contained in 
Releases 6 and 7. 

The following chart summarizes the function point counts for Releases 1 and 2, 
Releases 3 and 4, and Releases 5 through 7 plus Selectivity 1 through 3. 
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Size of ACE Software by Release Groupings 

Releases 1 and 2 
5 percent 




Releases 5, 6, and 7 
Selectivity 1, 2, and 3 
86 percent 



Source: GAO based on CBP data. 
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Releases 1 , 2, 3, and 4 reportedly account for 1634 function points being 
addressed by custom software and COTS, including SAP. In contrast, Releases 5, 
6, and 7 plus Selectivity 1 , 2, and 3 constitute 10,302 function points, 5450 of which 
are to be addressed by custom software and other COTS with the balance of 4852 
to be addressed by the SAP product. The graph on the following page provides this 
information by release. 

Despite these relative sizes, the scheduled time for completing the last six releases 
is about the same as the time anticipated to complete the first four releases, even 
though the software size of the last six is over three times that of the first four, and 
this does not include SAP software. Specifically, as of January 2004, the total time 
projected for design, development, and deployment of Releases 1 to 4 is 
approximately 3 years. According to program documentation, the total time 
projected for design, development, and deployment of Releases 5, 6, and 7 and 
Selectivity 1, 2, and 3 is approximately 3-1/2 years. 

Program officials stated that they expect the Global Business Blueprint activity that 
they are currently performing to better identify the extent to which function points 
can be addressed by custom software, other COTS, and SAP in Releases 5, 6, and 
7 and Selectivity 1 , 2, and 3. 
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The following graph illustrates the amount of custom software, other COTS, and 
SAP software associated with each release: 1 




□ SAP software 

□ Other COTS 

□ Custom software 

■ Custom and 
COTS 



Releases Releases Release 5 Selectivity Release 6 Selectivity Release 7 Selectivity 
1-2 3-4 1 2 3 

Source: GAO based on eCP data 

1 Releases 1 to 4 function points were not allocated between custom, other COTS, and SAP; however, 
these releases consist of custom and COTS software, including SAP. 
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Observation 7: Reliability of key cost estimates in expenditure plan is unclear. 

Cost estimates in expenditure plan varied from independent cost estimates. 

CBP has established a process for developing cost estimates in expenditure plans. 
This process calls for estimates from both eCP and from an independent cost 
estimating contractor. The process also calls for reconciling these estimates before 
cost estimates are incorporated into an expenditure plan. 

In formulating the fiscal year 2004 ACE expenditure plan, CBP obtained eCP cost 
estimates in early October 2003, and most independent cost estimating contractor 
estimates in late October 2003. However, CBP used the eCP estimates in the 
expenditure plan without reconciling them to those of the independent cost estimator. 
According to program officials, the reconciliation step was not done because CBP 
did not have sufficient time to do so before completing the plan and submitting it for 
DHS review in mid-October 2003. However, CBP did task its support contractor with 
reconciling the estimates, including analyzing the reasons for differences. This 
reconciliation was completed in mid-November 2003, about 2 months before the 
expenditure plan was submitted to the appropriations committees. The net difference 
between eCP's estimates in the plan and the independent estimates is $3.4 million. 
However, the sum of the differences for individual expenditure plan elements is 
$39.8 million, as shown on the following table. 
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Comparison of expenditure plan cost estimates to independent estimates 

Difference between 
Expenditure Plan Cost 
Estimates and Independent 
Cost Estimates 



Element 1 


Net 


Total 


Explanation of difference 


Release 6 definition and 


0.4 


0.4 


eCP used a broader interpretation of the Release 6 scope 


design 






than the independent estimator. 


Selectivity 2 definition 
and design 


2.7 


2.7 


eCP used a 21 percent higher software size estimate and 9 
percent higher assumed labor rate than the independent 
estimator. 


Implementation 
infrastructure and support 








N/A 


Foundation program 
management 


-4.4 


4.4 


Cost data reflecting previous experience with this activity were 
not incorporated into the independent estimate. 


Foundation architecture 


7.8 


7.8 


Scope was not defined, and the movement of 1 to 13 


and engineering 






positions to this activity was not identified in the independent 
estimate. 


Workforce transformation 


10.7 


10.7 


eCP used a broader view of the scope of the workforce 


and training 






transformation task statement than the independent estimator. 


Operations and 


-13.8 


13.8 


Independent cost estimate used a broader view of hardware 


maintenance 






and software maintenance costs. 


Total 


3.4 


39.8 





Source: GAO analysis based on CBP data. 



Additional expenditure plan elements (i.e., CBPMO, Federally Funded Research and Development Center, International Trade Data System, SAP, and 
management reserve) were not estimated by the independent estimator and thus not included in this analysis. 
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Expenditure plan cost estimates partially satisfy relevant guidance. 

One indicator of the reliability of cost estimates is the effectiveness of processes 
used to derive the estimates. SEI has defined guidelines for assessing such 
processes and determining the reliability of estimates. 1 

To determine the reliability of the cost estimates in the expenditure plan, the 
support contractor and eCP analyzed the estimates against SEI guidance. The cost 
estimates analyzed pertained to the following plan elements: Release 6 definition 
and design, Selectivity 2 definition and design, foundation program management, 
architecture and engineering, workforce transformation and training, and operations 
and maintenance. The analysis shows that the expenditure plan estimates partially 
satisfy the SEI's guidance. The results of this analysis follow. 
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1 SEI's institutional estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost and Schedule 
Estimating Capabilities of Software Organizations, and project-specific estimating guidelines are defined in A Manager's 
Checklist for Validating Software Cost and Schedule Estimates. 
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SEI guidance Satisfied Assessment of criteria satisfaction 

1 . Are the objectives of the Yes a According to the support contractor, the objective of the 

estimate clear and correct? estimate was clearly stated. Also, the tasks and activities to be 

included in, and excluded from, the estimate were clearly 
identified. 

According to the support contractor, software sizing estimates 
were determined by requirements and function point count 
performed by certified staff. However, the process for estimating 

the size of the task was not adequately documented. 

According to the support contractor, the cost and schedule 
models used to develop the estimate were calibrated to eCP's 
historical data on other programs and demonstrated experience 
from release 2. However, eCP has not demonstrated how its 
estimating models take into account demonstrated performance 
across different system development methodologies (e.g., 

custom development versus SAP implementation). 

4. Have the factors that affect Partially According to the support contractor, parameter values and 
the estimate been identified planning assumptions were documented. However, 

and explained? uncertainties in parameter values were not identified. 

Source: GAO. 

a Satisfies every aspect of the condition that we reviewed. 

b Satisfies many, but not all, key aspects of the condition that we reviewed. 
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appropriately sized? 



3. Are the estimated cost and Partially 
schedule consistent with 
demonstrated 
accomplishments on other 
projects? 
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Objective 3 Results 
Observation 7 


SEI Guidance 

V^^— 1 muiviui IVv 


Satisfied 

V^U llw 1 IVM 


Assessment of criteria satisfaction 


5. Have steps been taken 
to ensure the integrity 
of the estimating 
process? 


Partially 


According to the support contractor, eCP set up a planning team 
that includes certified analysts and experienced estimators. 
However, eCP's schedule estimating process was not 
adequately documented, the cost models do not support 
schedules with the extensive level of parallel development 
activities that are necessary to achieve the proposed schedule, 
and the shift of requirements from Release 2 to future releases 
may not be accurately reflected in the estimate. 


6. Is the organization's 
historical evidence 
capable of supporting 
a reliable estimate? 


Partially 


According to the support contractor, eCP draws from a database 
that contains productivity data and supports cost percentages 
from a large set of completed projects. However, cost models did 
not have consistent methods for recording historical data, and 
eCP has not demonstrated how it measures and improves its 
software processes based on historical evidence. 


7. Has the situation 
changed since the 
estimate was 
prepared? 


Yes 


According to the support contractor, program changes (e.g., 
schedule changes) were incorporated in estimates. Additionally, 
the estimate was used as the basis for assigning resources, 
deploying relevant schedules, and making commitments. 


Source: GAO. 
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Accountability * Integrity * Reliability 



Conclusions 



The fiscal year 2004 ACE expenditure plan, along with program documentation and 
officials' commitments, largely satisfies the legislative conditions imposed by 
Congress. Further, many of the recommendations that we have previously made to 
strengthen ACE management have been addressed, and CBP leadership has 
committed to addressing the remaining recommendations. However, much remains 
to be done before these open recommendations are fully implemented. In 
particular, progress on overcoming human capital challenges has been slow. 

To CBP's credit, the first two ACE releases are operating largely as intended. 
Achieving this initial operating capability, however, has not occurred without some 
difficulty, such as borrowing resources from future releases to overcome problems 
on near-term releases; similar difficulties that could impact cost, schedule, and 
capability commitments are being experienced on ongoing releases. Prospectively, 
this pattern is not likely to change unless the degree of concurrency among 
activities within and between releases is better controlled, and the underlying 
reasons for introducing this concurrency are addressed. While time and resources 
were invested in management improvements over a year ago to address the root 
causes, the absence of meaningful measures for determining whether these 
investments will be successful, coupled with limited progress on some of our 
previous recommendations, makes the chances of CBP delivering future ACE 
release capabilities on time and within budget uncertain at best. 
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Recommendations 



To assist CBP in managing ACE and increasing the chances that future releases 
will deliver promised capabilities on time and within budget, we recommend that the 
DHS Secretary, through the Under Secretary for Border and Transportation 
Security and the CBP Commissioner, direct the CBP CIO to 

• ensure the independence of its IV&V agent; 

• ensure that future expenditure plans are based on cost estimates that are 
reconciled with independent cost estimates; 

• reconsider the ACE acquisition schedule and cost estimates in light of early 
release problems, including these early releases' cascading effects on future 
releases and their relatively small size compared to later releases, and in light 
of the need to avoid the past levels of concurrency among activities within and 
between releases; 

• define measures, and collect and use associated metrics, for determining 
whether prior and future program management improvements are successful; 
and 

• report quarterly to the House and Senate Appropriations Committees on 
efforts to address the above, as well as our previously made but still open 
recommendations. 
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Accountability * Integrity * Reliability 



Agency Comments 



We provided this briefing to and discussed its contents with DHS, CBP, and ACE 
program officials. These officials, including the DHS Chief Information Officer and 
the CBP Acting Chief Information Officer, generally agreed with our findings, 
conclusions, and recommendations, and stated that the briefing was fair and 
balanced. They also provided some technical comments, which we have 
incorporated into the briefing, as appropriate. 
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Scope and Methodology 



Scope and Methodology 

To accomplish our objectives, we analyzed the ACE fiscal year 2004 expenditure 
plan and supporting documentation, comparing them to relevant federal 
requirements and guidance and to applicable best practices; and we interviewed 
DHS and CBP officials and ACE program contractors. In particular, we reviewed 

• DHS and CBP investment management practices, using OMB A-1 1 , part 7; 

• DHS and CBP activities for assessing ACE compliance with the DHS 
enterprise architecture; 

• CBP acquisition management practices, using the Software Engineering 
Institute's (SEI) Software Acquisition Capability Maturity Model (SA-CMM); 1 



1 The Software Acquisition Capability Maturity Model® developed by Carnegie Mellon University's Software Engineering 
Institute (SEI) defines five levels of maturity that provide a roadmap for continuously improving an organization's acquisition 
management processes. 
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A Q ^ Q Attachment 1 



Accountability * Integrity * Reliability 



Scope and Methodology 



CBP cost estimating program, using SEI's institutional and project-specific 
estimating guidelines; 1 

CBP human capital management activities, against its human capital 
management strategy; 

ACE testing activities, system defect data, and system performance data 
using industry best practices; 2 

independent verification and validation (IV&V) activities using the Institute of 
Electrical and Electronics Engineers Standard for Software Verification and 
Validation; 3 



1 SEI's institutional estimating guidelines are defined in Checklists and Criteria for Evaluating the Cost and Schedule 
Estimating Capabilities of Software Organizations, and SEI's project-specific estimating guidelines are defined in A Manager's 
Checklist for Validating Software Cost and Schedule Estimates. 

2 U.S. General Accounting Office, Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21 (Washington, D.C.: 
November 1998). 

institute of Electrical and Electronics Engineers (IEEE) Standard for Software Verification and Validation, IEEE Std 1012- 
1998 (New York: Mar. 9, 1998). 
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A Q ^ Q Attachment 1 



Accountability * Integrity * Reliability 



Scope and Methodology 



• cost and schedule data, and functional requirements from program 
management documentation; and 

• CBP and its contractor activities for improving the ACE program with program 
improvement action plans. 

We did not independently verify DHS-, CBP-, and contractor-provided data or 
calculations nor do we attest to their accuracy. 

For DHS-, CBP-, and contractor-provided data that we did not substantiate, we 
have made appropriate attribution indicating the data's sources. 

We conducted our work at CBP headquarters and contractor facilities in the 
Washington, D.C., metropolitan area from January through March 2004 in 
accordance with generally accepted government auditing standards. 
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U.S. Department of Homeland Security 

Washington, DC 20528 



Homeland 
%ggp Security 

DATE: April 30, 2004 



MEMORANDUM FOR RANDOLPH C. HUE 

U.S. GENERAL ACCOUNTING OFFICE 



FROM: 



SUBJECT: 



Anna F. Dixon 

Director, Bankcard Programs and 
GAO/OIG Liaison 

General Accounting Office (GAO) Draft Report: 
Information Technology: Early Releases of 
Customs Trade System Operating, but Pattern 
of Cost and Schedule Problems Needs to be 
Addressed (GAO-04-587) 



Thank you for providing us with a copy of your draft report and the opportunity to 
discuss the issues in this report. 

U.S. Customs and Border Protection (CBP) either has underway or has completed 
actions responsive to the recommendations in the report. Responses to the specific 
recommendations are included as Attachment 1 . 

Additionally, it is CBP's recommendation that specific budget line item amounts 
contained in the Fiscal Year 2004 Modernization Expenditure Plan be excluded from the 
final version of the GAO report. Continuing to report the total expenditure plan funding 
amount (31 8M) is acceptable and does not impact upcoming contract negotiations. 
Specific edits to the draft GAO report to resolve this issue are included as Attachment 2. 

If you have any questions regarding these comments, please contact Ms. Michele 
Donahue at (202) 927-0957. 



^4-^7 Anna F. Dixon 



Director, Bankcard Programs and 
GAO/OIG Liaison 



Attachments as stated 



www.dhs.gov 
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U.S. Customs Service 
General Accounting Office (GAO) Review of Fifth Automated Commercial 
Environment (ACE) Expenditure Plan 



To assist CBP in managing ACE and increasing the chances that future releases 
will deliver promised capabilities on time and within budget, we recommend that 
the DHS Secretary, through the Under Secretary for Border and Transportation 
Security and the CBP Commissioner, direct the CBP CIO to: 

Recommendation 1 : Ensure the independence of its IV&V agent 

Response: 

CBP will address GAO's concern about the independence of its IV&V agent by 
clarifying that management and financial oversight of the agent is separate from 
the Customs and Border Protection Modernization Office (CBPMO) and rests 
with the Special Assistant for Audit and Quality Management. 

Milestone Date: May 31 , 2004 



Recommendation 2: Ensure that future expenditure plans are based on 
cost estimates that are reconciled with independent cost estimates. 

Response: 

The CBPMO's Expenditure Plan Development and Approval Process has been 
updated to reflect a more significant role for CBPMO's independent cost 
estimating (ICE) contractor. The ICE contractor will be actively involved in the 
development process and will be the primary source of cost, schedule and risk 
input. 

Milestone Date: July 1 , 2004 
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Attachment 1 Attachment 1 

-2- 



Recommendation 3: Reconsider the ACE acquisition schedule and cost 
estimates in light of early release problems, including these early releases' 
cascading effects on future releases and their relatively small size compared to 
later releases, and in light of the need to avoid the past levels of concurrency 
among activities within and between releases. 

Response: 

CBP will respond to this recommendation through the current effort to develop 
the next ACE program plan. 

Milestone Date: July 31 , 2004 



Recommendation 4: Define measures, and collect and use associated 
metrics, for determining whether prior and future program management 
improvements are successful. 

Response: 

CBP has taken recent steps responsive to this recommendation. CBPMO has 
developed a metrics plan that is currently undergoing final review. This plan 
draws upon measures that have been used as well as incorporating new metrics 
into a balanced scorecard to enable analysis within CBP and with eCP regarding 
program performance trends. 

Milestone Date: May 15, 2004 for final plan approval 



Recommendation 5: Report quarterly to the House and Senate 
Appropriations Committees on efforts to address the above, as well as our 
previously made but still open recommendations. 

Response: 

CBP will respond to this recommendation by continuing its practice of 
incorporating status reports on its progress addressing GAO open 
recommendations in its quarterly reports to the Appropriations and Authorization 
Committees. 

Milestone Date: Considered closed based upon prior congressional reports. 
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ATTACHMENT 2 ATTACHMENT 2 



Pages 29 and 84 of the draft GAO report contain specific budget information that, 
if published, could weaken the government's negotiating position in a number of 
upcoming contract actions that total approximately $60 million. The CBP 
recommends removing the detailed budget line data as illustrated in the modified 
versions of pages 29 and 84 below and adding the appropriate disclaimer in the 
footnote of each page. 



Sensitive but Unclassified 

- A O Background 

e»P- vJ x\ \J Summary of Expenditure Plan Funding 



Summary of the ACE fiscal year 2004 expenditure plan 

Plan activity Funding* 

Release 6 definition and design 

Selectivity 2 definition and design 

Implementation infrastructure and support 

Foundation program management 

Foundation architecture and engineering 

Workforce transformation and training 

Operations and maintenance 

CBFMO costs 

Federally Funded Research and Development Center 

International Trade Data System 

SAP* 



Management reserve 



Total 



Source: QAO based on CBP data. 



" Mffions of dollars. Specific budget ine information has been excluded to preserve the government's negotiating position. 
6 SAP is a commercial enterprise resource planning software product that has multiple modules, each performing separate 
but integrated business functions. CBP is using SAP as part of a joint project between its Office of Finance to support 
financial and property management and *>e CBPMO to support such ACE functions as customer relationship management. 



29 



L GAO 



Sensitive but Unclassified 



Objective 3 Results 
Observation 7 



Comparison of expenditure plan cost estimates to independent estimates 


Element' 


Expenditure 
plan cost 
estimate 1 


Independent Difference 
cost estimate 1 Net Total 


Explanation of difference 


Release 6 definition and 
design 




0.4 


0.4 


eCP used a broader interpretation of the 
Release 6 scope than the independent 
estimator. 


Selectivity 2 definition 
and design 




2.7 


2.7 


eCP used a 21 percent higher software size 
estimate and 9 percent higher assumed 
labor rate than the independent estimator. 


Implementation 
infrastructure and support 










N/A 


Foundation program 
management 




-4.4 


4.4 


Cost data reflecting previous experience with 
this activity were not incorporated into the 
independent estimate. 


Foundation architecture 
and engineering 




7.8 


7.8 


Scope was not defined, and the movement of 
10 to 13 positions to this activity was not 
identified in the independent estimate 


Workforce transformation 
and training 




10.7 


10.7 


eCP used a broader view of the scope of the 
workforce transformation task statement than 
the independent estimator. 


Operations and 
maintenance 




-13.8 


13.8 


Independent cost estimate used a broader 
view of hardware and software maintenance 
costs 


Total 


244.7 


241.3 3.4 


39.8 




e^pSrfihj'r*' r*^l6menH (i.e., CBPMO, Federally Funded Research 
management reserve) ware not estirna&d by the independent estimator and thus 
2 Specific budget ine information has been excluded to preserve the govemmertf 


and Development Center, tntematjonal Trade Data System, SAP, and 
not ncluded in ttm analysis, 
t negotiating position. 
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The General Accounting Office, the audit, evaluation and investigative arm of 
Congress, exists to support Congress in meeting its constitutional responsibilities 
and to help improve the performance and accountability of the federal government 
for the American people. GAO examines the use of public funds; evaluates federal 
programs and policies; and provides analyses, recommendations, and other 
assistance to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core values of 
accountability, integrity, and reliability. 
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products. The Web site features a search engine to help you locate documents 
using key words and phrases. You can print these documents in their entirety, 
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correspondence. GAO posts this list, known as "Today's Reports," on its Web site 
daily. The list contains links to the full-text document files. To have GAO e-mail this 
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e-mail alerts" under the "Order GAO Products" heading. 
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